Papers by Leo Yu Zhang
Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are vulnerable to data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. |
| Approach: | They propose an Optimal Transport-based framework to reconstruct image-caption pairs and propose an optimal transport-based distance measure to re-assign new captions based on the proposed optimal transport distance. |
| Outcome: | The proposed framework reduces the attack success rates of poisoning attacks to 0% in most cases. |